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1500 


microservices 


Written on different 
languages 
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2000+ 5 


engineers K8S clusters 
With different skills, In different data 
grade, experience. centers, in different 
Working on different parts of the country 


business domains 


With the growth in the number of services, the role of 
documentation grows 


avito.tech 


WHY IT'S IMPORTANT 


Easy integration Team 
between services onboarding 
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Much safer to make 
diffs in other teams 
services 


In fact there are much more bonuses... 
But we will talk about later... 
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WHAT IS — GOOD DOCUMENTATION 


э 


avito.tech 


ON INTERVIEW 


Ok, after the interview questions 
You can ask me anything about 
company and work there. 


You Candidate 
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ON INTERVIEW 


Do you have documentation on 
your projects? 

Is it good? 

Can I use it? 

Will it help me to onboard? 


Candidate 
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ON INTERVIEW 


Candidate 
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Perhaps you see some strange guy, 
asking about documentation in 
different chats and topics.... 


That was me 
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We need some analysis...and a bit of group by 


DESCRIPTION 


You want to integrate with the service 


Name, Description 
Feature Toggles 


Owner, team, duty 


v YV YV Nw 


QualityGates 
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QUALITY GATES 


i 


It will help us understand Examples: 

the cost of integration 

and predict quality of out » Test Coverage 
design » Required e2e tests 


№ Linter status 
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INTEGRATION 


p Service API 


» Integration guide 


p Service NFR + SLOVSLA 
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Non-functional 
Requirements 


Describe every 
handler in 

artefact 

- Max\Min RPM 

- Latency 

- Expected Error Rate 
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error codes 


name 


type = REST" 
latency — '100ms" 


error codes 


SERVICE API 


If you need to rich existing service 
answer, service API will help to 
understand how to do it efficiently 
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Swagger Ч 


Veeam Backup & Replication REST API =P €» 


Select a definition aa | 


/api/oauth2/token Get Access Token 


/api/oauth2/authorization_code Get Authorzation Code 


/api/oauth2/logout Log Out 


fapi/vi/serverTime Get Sever Time 


/api/v1/servercertificate 


ШС /api/vastrafficrules Est Trate Rules 


General Options The селе 


IE ET 
| 
(ШЕГЕ api /va/generaloptions Est Gener Otons 


Configuration Backup т^ congue 
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DEVELOP 


Product description 
Dependencies and Consumers API 
Deploy plan 

Integration Guide 


SLO\SLA\NFR 


v Y YV vv v 


TDR\C4 Model 
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PRODUCT DESCRIPTION 


O].  Whythis service exists? 03. Who use it? What Platforms? 
On what screens? 


(02, What problems it solves 04. What happens if derive will 
down? Does it use on user crit 
scenarios? 
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Dependencies and Consumers API 


> Help with troubleshoot 
» Very helpful for on-call 


> Architecture optimisation 
(analysis) 
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DEPLOY PLAN AND ROLLBACK 
gm, 


> Rules and conditions of deploy 


» Places for deploy 
» Required resources 


» Troubles and nuances of deploy 
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C4-MODEL 


You can use any approach for 
visualisation of architect design 


We use: 
» C4 Model 


» Architecture Design Record 
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The C4 model for visualising 
ез software architecture 


c4model.com 


Zoom in 


Zoom in 


Zoom in 


Level 1 Level 2 Level 3 Level 4 
Context Containers Components Code 


https://c4model.com 
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RUNTIME 


You are on-call engineer, and you need 
some kind of service troubleshoot 


Service observability links 
Latest releases 


Dependencies and Consumers 


v v YV Yy 


Rollback plan 
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SERVICE OBSERVABILITY 


Links to Logs 
(Kibana, 
Sentry) 
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Main metrics: 
RPM, 

Latency 
Percentile 


Traces (open 
telemetry with 
Jaeger) 


Links to ALERTS 


Alerts.yaml in git 
for every service 


triggers: 
- name: Panics on service 
id: 5f5f0e45-9b74-42ac-91f4-09eb25a5b1d3 
desc: >- 
Instruction - how this problem can be solved 
targets: 
- aliasByNode(graphiteBucke.api.panic.total, -2, -1) 
ttl_state: 0K 
ЕЕС: 120 
expression: "(tl > 2) ? ERROR : OK" 
dashboard: https://link-tografana-dashboard 
pending_interval: 60 


- name: 5xx in answers 
desc: >- 
Instruction - how this problem can be solved 
targets: 
- aliasByNode(movingAverage(sortByMaxima(exclude(keepLastValue(path-to-service.*.request time. 
(5x 4993. count; 1), “info GET')), "Imin'). 5, 7) 
ttl state: OK 
ttl: 600 
expression: "tl » 1000 ? ERROR : OK" 
dashboard: https://link-tografana-dashboard 
pending interval: 300 
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It's very important to do it all... 
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ON INTERVIEW 


Is it a good documentation? 


You Candidate 
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Different ways of documentation 
process, from service to service 


— 8 2 
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MAKE UNIFY SERVICE CARD 


avito.tech Make it look and feel unify for all services за 


BENEFITS 


Simple 
navigation 
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Good search 
between services 


Unify user experience 
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ON INTERVIEW 


Is it a good documentation? 


You Candidate 
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© 


Custom 
deploys is hard 
to support. 
Works 
different from 
service to 
service 
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g 


Metrics between 
services are not 
consistent! It's 
hard tom match 
metrics from 
one service to 
another 


Manually filled 
release list leads 
us to mistakes. 


Manually 
supported 
service API and 
links will be 
eventually 
inconsistent 
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Unification and automation for all processes! 
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UNIFY DEPLOY 
FLOW 


Let everyone service be deployed only in 
several configurations. And only on unify 
CD flow with simple customisation 


> Every release is simple, it's 
automatically register in release lists 


» You can implement different release 
strategies (gradual, canary, etc) 


> Every service will have unify rollback 
button 
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UNIFY METRICS 


Create standart metrics for rpc\rest\async 
requests. 

Implement «golden metrics» in standard 
buckets 


p Easy to create canary strategy 
p Unify alerts for similar services 


> Every service will have unify rollback 
button 
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ON INTERVIEW 


Is it a good documentation? 


You Candidate 
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Service 
Implementation 


| 2 
ханыг 


Contract 


MEGA-important to make contract system, based on 
contract-first principles! 


It helps your API-contract documentation part to be actual 
and consistent 
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CONTRACT FIRST: BENEFITS 


ОТ. 


02. 


Teams can develop new 
features in parallel, without 
lock on each other 


Teams gets exact they are 
waiting for. No missing field or 
typo errors 
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04. 


Codegen-part will help you 
make cross platform and 
cross language compatibility 


Scheme reuse helps decrease 
copy-paste and increase 
efficiency. 
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CONTRACT FIRST: LIMITATIONS 


ОТ. You should create standards 03, You will had to create 
for your internal and external contract update and validate 
API system 


02. It's expensive at the first time 
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CONTRACT 


o 


Your contract documentation will be always actual and 
full! 
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CONTRACT FIRST: CREATING CONTRACTS 


» Describe service new 
method in rpc.yml 


» Describe input, output 
schema for new method 


» Runcodegen command, for 
creating empty handler and 
to for new method in 
service 
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my-project/ 


api 


— data-bus 


I— consume 


| 
| 
| 
= 


L— servicel.yaml 


L— produce 


L— events.yaml 


ients 
— service2.yaml 
L— service3.yaml 
service 
L— rpc.yaml 
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CONTRACT FIRST: 


p New method will be 
register in schema-storage 


» Changes in old method will 
be validate for backward 
compatibility with 
consumers, using that 
method 


№» After success validation 


modify method will be 
updated in schema-storage 
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FIRST DEPLOY 


Validation 
Will check every deploy 


» All method consumers input 
against service schema 


» All consumers output against 
service schema 
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CONTRACT FIRST: FIRST DEPLOY 


rpc calcWords(input) output rpc calcWords(input) output 


intput = { intput = { 
sentencel: string, sentencel: string, 
sentence2: string = 
} } 
Check backward 
output = { Compatibility output = { 
cales tne cale: unt 


} 


SERVICE CONSUMER 
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CONTRACT FIRST: CONSUMER ADD DEP 


» Describe exact method, 
which needs. 


» Describe exact fields for 
method output 


» Вип codegen command, for 
creating empty handler and 
to for new method in 
service 
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my-project/ 


api 


— — data-bus 


— consume 


| L—— service1.yaml 
produce 


L— events.yaml 


rpe 


— clients 


| —— service2.yaml 


L — service3.vaml 
service 


L— rpc.yaml 


өөө 
rpc calcWords(input) output 


intput = { 
sentencel: string, 
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CONTRACT FIRST: CONSUMER DEPLOY 


» Validation newly added 


service өөө өөө 


rpc calcWords(input) output rpc calcWords(input) output 


» Describe exact fields for 
method output intput = { 


tntput = 1 


sentencel: string, sentencel: string, 


» Run codegen command, for : ВЕСЕ а —— 


creating empty handler and Consumer schema 
to for new method in output Should be subset 


service calc: int Of service 
error: sting 
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ON INTERVIEW 


Is it a good documentation? 


You Candidate 
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MkDocs 


A: 


Is part of code, 
described as 
usual markdown 
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Is deployed with 
service on 


codegen phase. 


Can be unite in one 
documentation for all 
service 


ON INTERVIEW 


YES! 
YOU ARE MY HERO! 


Candidate 
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OI. Now we have universal 
description for microservices 


O2. Now we have links to all 
significant runtime metrics 
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03. 


04. 


We have self-supported 
actual and fresh API 
documentation 


We have in-code generated 
documentation 
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Links: 


Leave your feedback! 


You can rate the talk and give 
a feedback on what you've 
liked or what could be 
improved 
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